[skills] Improve android-reviewer skill based on eval findings#11075
Merged
jonathanpeppers merged 3 commits intomainfrom Apr 3, 2026
Merged
[skills] Improve android-reviewer skill based on eval findings#11075jonathanpeppers merged 3 commits intomainfrom
jonathanpeppers merged 3 commits intomainfrom
Conversation
Changes to SKILL.md: - Add guidance that reviews should produce inline comments even on clean PRs (use suggestions for code consolidation, missing tests, etc.) - Add prioritized checklist for what to look for in diff analysis - Clarify CI verdict consistency: internal pipeline failures still require Needs Changes verdict - Document --dry-run flag for submit_review.cs Changes to review-rules.md: - Rename section 5 to 'Async, Cancellation & Thread Safety Patterns' - Add 'Double-checked locking requires volatile' rule with detailed ARM64 memory model guidance - Add 'Singleton initialization completeness' rule for publish-before- setup bugs Changes to submit_review.cs: - Add --dry-run flag that validates and pretty-prints review JSON without posting to GitHub - Use Utf8JsonWriter for AOT-compatible JSON serialization Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The volatile keyword is discouraged by Microsoft's own documentation. Update the DCL review rule to recommend Lazy<T> or LazyInitializer instead, with a link to the MS Learn docs. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refines the .github/skills/android-reviewer skill guidance and tooling based on evaluation feedback, aiming to improve review consistency (inline comments, CI verdict handling) and add a safer review submission workflow via a --dry-run mode.
Changes:
- Updated
SKILL.mdto encourage at least one inline suggestion on non-trivial PRs, add a prioritized diff-review checklist, and clarify internal CI failure verdict handling. - Expanded
review-rules.mdwith additional thread-safety guidance (avoid DCL; ensure singleton init completeness) and renamed the async section accordingly. - Enhanced
submit_review.cswith a--dry-runoption that pretty-prints validated review JSON usingUtf8JsonWriter.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| .github/skills/android-reviewer/SKILL.md | Adds guidance for inline comment expectations, a review priority checklist, CI verdict clarification, and documents --dry-run usage. |
| .github/skills/android-reviewer/references/review-rules.md | Renames async section and adds new thread-safety rules (avoid DCL; singleton init completeness). |
| .github/skills/android-reviewer/scripts/submit_review.cs | Adds --dry-run mode and uses Utf8JsonWriter to print validated JSON without posting. |
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
jonathanpeppers
deleted the
jonathanpeppers/evaluate-android-reviewer-skill
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Iterative improvements to the
android-reviewerskill based on structured evaluations against 4 real PRs (3 Copilot-authored, 1 human-authored).Changes
SKILL.md
--dry-runflag forsubmit_review.csreview-rules.md
Lazy<T>orLazyInitializer" rule with MS Learn referencesubmit_review.cs
--dry-runflag that validates and pretty-prints review JSON without posting to GitHubUtf8JsonWriterfor AOT-compatible JSON serialization (fixes crash on .NET 10)Evaluation Results
Tested against 4 PRs across 3 iterations:
Key improvements: